Aiming at the problem of the current common Convolutional Neural Network(CNN)in the expression and gender recognition tasks,that is training process is complicated,time-consuming,and poor in real-time performance,a realtime facial expression and gender recognition model based on depthwise separable convolutional neural network was proposed. Firstly,the Multi-Task Convolutional Neural Network(MTCNN)was used to detect faces in different scale input images,and the detected face positions were tracked by Kernelized Correlation Filter(KCF)to increase the detection speed. Then,the bottleneck layers of convolution kernels of different scales were set,the kernel convolution units were formed by the feature fusion method of channel combination,the diversified features were extracted by the depthwise separable convolutional neural network with residual blocks and separable convolution units,and the number of parameters was reduced to lightweight the model structure. Besides,real-time enabled backpropagation visualization was used to reveal the dynamic changes of the weights and characteristics of learning. Finally,the two networks of expression recognition and gender recognition were combined in parallel to realize real-time recognition of expression and gender. Experimental results show that the proposed network model has a recognition rate of 73. 8% on the FER-2013 dataset,a recognition rate of 96% on the CK+ dataset,the accuracy of gender classification on the IMDB dataset reaches 96%;and this model has the overall processing speed reached 70 frames per second,which is improved by 1. 5 times compared with the method of common convolutional neural network combined with support vector machine. Therefore,for datasets with large differences in quantity,resolution and size,the proposed network model has fast detection,short training time,simple feature extraction, and high recognition rate and real-time performance.